Method and system for communicating between memory regions

ABSTRACT

A method and system are provided for transferring data in a networked system between a local memory in a local system and a remote memory in a remote system. A RDMA request is received and a first buffer region is associated with a first transfer operation. The system determines whether a size of the first buffer region exceeds a maximum transfer size of the networked system. Portions of the second buffer region may be associated with the first transfer operation based on the determination of the size of the first buffer region. The system subsequently performs the first transfer operation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of prior co-pending U.S. patentapplication Ser. No. 12/542,546 filed Aug. 17, 2009, which is acontinuation of prior U.S. patent application Ser. No. 09/576,038 filedMay 23, 2000.

FIELD

The present invention generally relates to data networks and inparticular relates to a method and system for performing memory tomemory transfers between two systems.

BACKGROUND

A data network is generally consisted of a network of nodes connected bypoint-to-point links. Each physical link may support a number of logicalpoint-to-point channels. Each channel may be a bi-directionalcommunication path for allowing commands and message data to flowbetween two connected nodes within the data network. Each channel mayrefer to a single point-to-point connection where message data may betransferred between two endpoints or systems. Data may be transmitted inpackets including groups called cells from source to destination oftenthrough intermediate nodes.

In many data networks, hardware and software may often be used tosupport asynchronous data transfers between two memory regions, often ondifferent systems. Each system may correspond to a multi-processorsystem including one or more processors. Each system may serve as asource (initiator) system which initiates a message data transfer(message send operation) or a target system of a message passingoperation (message receive operation). Examples of such amulti-processor system may include host servers providing a variety ofapplications or services, and I/O units providing storage oriented andnetwork oriented I/O services.

Next Generation I/O (NGIO) architecture, virtual interface (VI)architecture and Infiniband architecture provide an I/O communicationmechanism among different computer systems. Communication allowsmovement of data between two memory regions, typically on differentsystems. Because hardware limitations exist with NGIO, VI and Infinibandarchitectures, a remote direct memory access (RDMA) operation may createa request larger than that supported by the underlying hardware andsoftware. More specifically, the RDMA request can transfer data frommultiple local memory buffers into a single remote memory region;however, the network hardware or software may impose limitations on thesize of a single data transfer. This may slow down the data transfer orresult in an incorrect data transfer. For example, many NGIOarchitectures can only support a 4 gigabyte data transfer due tohardware limitations. However, it may be desirable to transfer more than4 gigabytes of data efficiently.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and a better understanding of the present invention willbecome apparent from the following detailed description of exemplaryembodiments and the claims when read in connection with the accompanyingdrawings, all forming a part of the disclosure of this invention. Whilethe foregoing and following written and illustrated disclosure focuseson disclosing example embodiments of the invention, it should be clearlyunderstood that the same is by way of illustration and example only andis not limited thereto. The spirit and scope of the present inventionbeing limited only by the terms of the appended claims.

The following represents brief descriptions of the drawings in whichlike numerals relate to like elements and wherein:

FIG. 1 is a diagram illustrating an example data network having severalnodes interconnected by corresponding links of a basic switch accordingto an example embodiment of the present invention;

FIG. 2 illustrates another example data network having several nodesinterconnected by corresponding links of a multi-stage switched fabricaccording to an example embodiment of the present invention;

FIG. 3 illustrates a block diagram of a host system of a data networkaccording to an example embodiment of the present invention;

FIG. 4 illustrates a block diagram of a host system of a data networkaccording to another example embodiment of the present invention;

FIG. 5 illustrates an example software driver stack of a host operatingsystem of an example data network according to an example embodiment ofthe present invention;

FIG. 6 shows a RDMA data transfer request structure according to anexample embodiment of the present invention;

FIG. 7 shows RDMA coalescing and dividing and data transfer from thelocal memory to the remote memory according to an example embodiment ofthe present invention;

FIG. 8 shows RDMA coalescing and dividing according to an exampleembodiment of the present invention; and

FIG. 9 shows a RDMA request control overflow overview according to anexample embodiment of the present invention.

DETAILED DESCRIPTION

Before beginning a detailed description of the present invention,mention of the following is in order. When appropriate, like referencenumerals and characters may be used to designate identical,corresponding or similar components in differing figure drawings.Furthermore, in the detailed description to follow, examplesizes/models/values/ranges may be given, although the present inventionis not limited to the same. Whereas specific details are set forth inorder to describe example embodiments of the invention, it should beapparent to one skilled in the art that the invention can be practicedwithout these specific details. It should also be apparent that anycombination of hardware and software instructions can be used toimplement the embodiments of the present invention, i.e., the presentinvention is not limited to any specific combination of hardware andsoftware instructions.

The present invention may relate to NGIO, VI and/or Infinibandarchitectures that allow movement of data from multiple, local memorylocations to a single remote memory region. Other examples of computernetworks that may include the present invention are a local area network(LAN), a wide area network (WAN), a campus area network (CAN), ametropolitan area network (MAN), a global area network (GAN) and asystem area network (SAN). However, for the sake of simplicity,discussions will concentrate mainly on data movement in a simple datanetwork having example nodes (e.g., computers, servers and I/O units)interconnected by corresponding links and switches, although the scopeof the present invention is not limited thereto.

The present invention may provide a method and system by which multipleRDMA requests to a single, remote memory buffer can be merged (orcoalesced) together into one operation to thereby provide a betterperformance system through optimal resource utilization. However, singleor coalesced RDMA operations may create a RDMA request larger than theunderlying hardware and/or software can support. These requests may bedivided into smaller, individual blocks. This facilitates rapid driverdevelopment to support RDMA transfer requests larger than that providedby the underlying hardware and/or software. This also ensures thatcoalesced operations can be as efficient as possible.

Attention now is directed to the drawings and particularly to FIG. 1, inwhich a simple data network 10 having several interconnected nodes fordata communications according to an embodiment of the present inventionis illustrated. As shown in FIG. 1, the data network 10 may include, forexample, one or more centralized switches 100 and four different nodesA, B, C, and D.

The remainder of this disclosure will discuss a local computer systemand a remote computer system. For the sake of example, a local computersystem 20 may be provided at node A and a remote computer system 30 maybe provided at node D. The terminology local and remote is usedhereinafter to describe two systems that are physically separated fromeach other. The terminology local and remote does not imply anydifferences between the systems other than they are physically separatedfrom each other.

Each of the local system 20 and the remote system 30 may include one ormore input/output units (I/O units) and one or more I/O controllers.Each I/O controller may operate to control one or more I/O devices, suchas storage devices (e.g., a hard disk drive or tape drive) locally orremotely via a local area network (LAN) or a wide area network (WAN),for example.

The centralized switch 100 may contain, for example, switch ports 0, 1,2, and 3 each connected to a corresponding node of the four differentnodes A, B, C, and D via a corresponding physical link 110, 112, 114,and 116. Each physical link may support a number of logicalpoint-to-point channels. Each channel may be a bi-directionalcommunication path for allowing commands and data to flow between twoconnect nodes (e.g., host systems, switch/switch elements, and I/Ounits) within the network.

Each channel may refer to a single point-to-point connection where datamay be transferred between endpoints (e.g., host systems and I/O units).The centralized switch 100 may also contain routing information using,for example, explicit routing and/or destination address routing forrouting data from a source node (data transmitter) to a target node(data receiver) via corresponding link(s), and re-routing informationfor redundancy.

The specific number and configuration of end stations (e.g., hostsystems and I/O units), switches and links shown in FIG. 1 are providedsimply as an example data network. A wide variety of implementations andarrangements of a number of end stations (e.g., host systems and I/Ounits), switches and links in all types of data networks may bepossible.

According to an example embodiment or implementation, the end stations(e.g., host systems and I/O units) of the example data network shown inFIG. 1 may be compatible with the “Next Generation Input/Output (NGIO)Specification” as set forth by the NGIO Forum on Mar. 26, 1999.According to the NGIO Specification, the switch 100 may be an NGIOswitched fabric (e.g., collection of links, switches and/or switchelements connecting a number of host systems and I/O units), and theendpoint may be a host system including one or more host channeladapters (HCAs), or a target system such as an I/O unit including one ormore target channel adapters (TCAs). Both the host channel adapter (HCA)and the target channel adapter (TCA) may be broadly considered as fabricadapters provided to interface endpoints to the NGIO switched fabric,and may be implemented in compliance with “Next Generation I/O LinkArchitecture Specification: HCA Specification, Revision 1.0” as setforth by NGIO Forum on Jul. 20, 1999 for enabling the endpoints (nodes)to communicate to each other over an NGIO channel(s).

For example, FIG. 2 illustrates an example data network 10′ using a NGIOarchitecture to transfer data from a source node to a destination nodeaccording to an embodiment of the present invention. As shown in FIG. 2,the data network 10′ includes an NGIO fabric 100′ (multi-stage switchedfabric comprised of a plurality of switches) for allowing a host systemand a remote system to communicate to a large number of other hostsystems and remote systems over one or more designated channels. Asingle channel may be sufficient but data transfer speed betweenadjacent ports can decrease latency and increase bandwidth. Therefore,separate channels for separate control flow and data flow may bedesired. For example, one channel may be created for sending request andreply messages. A separate channel or set of channels may be created formoving data between the host system and any one of the target systems.In addition, any number of end stations, switches and links may be usedfor relaying data in groups of cells between the end stations andswitches via corresponding NGIO links.

For example, node A may represent the host or local system 20 such as ahost computer or a host server on which a variety of applications orservices are provided. Similarly, node B may represent another network150, including, but are not limited to, local area network (LAN), widearea network (WAN), Ethernet, ATM and fibre channel network, that isconnected via high speed serial links. Node C may represent an I/O unit170, including one or more I/O controllers and I/O units connectedthereto. Likewise, node D may represent the remote system 30 such as atarget computer or a target server on which a variety of applications orservices are provided. Alternatively, nodes A, B, C, and D may alsorepresent individual switches of the multi-stage switched fabric 100′which serve as intermediate nodes between the host system 20 and theremote systems 150, 170 and 30.

The multi-state switched fabric 100′ may include a central networkmanager 250 connected to all the switches for managing all networkmanagement functions. However, the central network manager 250 mayalternatively be incorporated as part of either the host system 20, thesecond network 150, the I/O unit 170, or the remote system 30 formanaging all network management functions. In either situation, thecentral network manager 250 may be configured for learning networktopology, determining the switch table or forwarding database, detectingand managing faults or link failures in the network and performing othernetwork management functions.

A host channel adapter (HCA) 120 may be used to provide an interfacebetween a memory controller (not shown) of the local system 20 and amulti-stage switched fabric 100′ via high speed serial NGIO links.Similarly, target channel adapters (TCA) 140 and 160 may be used toprovide an interface between the multi-stage switched fabric 100′ and anI/O controller of either a second network 150 or an I/O unit 170 viahigh speed serial NGIO links. Separately, another target channel adapter(TCA) 180 may be used to provide an interface between a memorycontroller (not shown) of the remote system 30 and the multi-stageswitched fabric 100′ via high speed serial NGIO links. Both the hostchannel adapter (HCA) and the target channel adapter (TCA) may bebroadly considered as fabric hardware adapters provided to interfaceeither the local system 20 or any one of the target systems 150, 170 and30 to the switched fabric, and may be implemented in compliance with“Next Generation I/O Link Architecture Specification: HCA Specification,Revision 1.0” as set forth by NGIO Forum on Jul. 20, 1999 for enablingthe endpoints (nodes) to communicate to each other over an NGIOchannel(s). However, NGIO is merely one example embodiment orimplementation of the present invention, and the invention is notlimited thereto. Rather, the present invention may be applicable to awide variety of any number of data networks, hosts and I/O units. Thepresent invention is also applicable to VI and Infiniband architectures.

One example embodiment of a host or local system 20 is shown in FIG. 3.Referring to FIG. 3, the host or local system 20 may correspond to amulti-processor system, including one or more processors 202A-202Ncoupled to a host bus 203. Each of the multiple processors 202A-202N mayoperate on a single item (I/O operation), and all of the multipleprocessors 202A-202N may operate on multiple items (I/O operations) on alist at the same time. An I/O and memory controller 204 (or chipset) maybe connected to the host bus 203. A main memory 206 may be connected tothe I/O and memory controller 204. An I/O bridge 208 may operate tobridge or interface between the I/O and memory controller 204 and an I/Obus 205. Several I/O controllers may be attached to the I/O bus 205,including an I/O controllers 210 and 212. I/O controllers 210 and 212(including any I/O devices connected thereto) may provide bus-based I/Oresources.

One or more host-fabric adapters 120 may also be connected to the I/Obus 205. Alternatively, one or more host-fabric adapters 120 may beconnected directly to the I/O and memory controller (or chipset) 204 toavoid the inherent limitations of the I/O bus 205 as shown in FIG. 4. Ineither embodiment, one or more host-fabric adapters 120 may be providedto interface the local system 20 to the multi-stage switched fabric100′.

FIGS. 3-4 merely illustrate example embodiments of a local system 20. Awide array of processor configurations of such a local system 20 may beavailable. A software driver stack for the host-fabric adapter 120 mayalso be provided to allow the local system 20 to exchange data with oneor more remote systems 150, 170 and 30 via the switched fabric 100′,while preferably being compatible with many currently availableoperating systems, such as Windows 2000.

FIG. 5 illustrates an example software driver stack of a host system 20.As shown in FIG. 5, a host operating system (OS) 500 may include akernel 510, an I/O manager 520, and a plurality of channel drivers530A-530N for providing an interface to various I/O controllers. Such ahost operating system (OS) 500 may be Windows 2000, for example, and theI/O manager 520 may be a Plug-n-Play manager.

In addition, a host-fabric adapter software stack (driver module) may beprovided to access the switched fabric 100′ and information about fabricconfiguration, fabric topology and connection information. Such ahost-fabric adapter software stack (driver module) may include a fabricbus driver 540 and a fabric adapter device-specific driver 550 utilizedto establish communication with a remote fabric-attached agent (e.g.,I/O controller), and perform functions common to most drivers,including, for example, host-fabric adapter initialization andconfiguration, channel configuration, channel abstraction, resourcemanagement, fabric management service and operations, send/receive I/Otransaction messages, remote direct memory access (RDMA) transactions(e.g., read and write operations), queue management, memoryregistration, descriptor management, message flow control, and transienterror handling and recovery. Such software driver module may be writtenusing high-level programming languages such as C, C++ and Visual Basic,and may be provided on a computer tangible medium, such as memorydevices; magnetic disks (fixed, floppy, and removable); other magneticmedia such as magnetic tapes; optical media such as CD-ROM disks, or viaInternet downloads, which may be available for a fabric administrator toconveniently plug-in or download into an existing operating system (OS).Such a software driver module may also be bundled with the existingoperating system (OS) which may be activated by a particular devicedriver.

The host-fabric adapter driver module may consist of three functionallayers: a HCA services layer (HSL), a HCA abstraction layer (HCAAL), anda HCA device-specific driver (HDSD) in compliance with the “NextGeneration I/O Architecture: Host Channel Adapter SoftwareSpecification.” For example, the HCA service layer (HSL) may be inherentto all channel drivers 530A-530N for providing a set of common fabricservices in a service library, including connection services, resourceservices, and HCA services required by the channel drivers 530A-530N toinstantiate and use NGIO channels for performing data transfers over theNGIO channels. The fabric bus driver 540 may correspond to the HCAabstraction layer (HCAAL) for managing all of the device-specificdrivers, controlling shared resources common to all HCAs in a host andresources specific to each HCA in the local system 20, distributingevent information to the HSL and controlling access to specific devicefunctions. Likewise, the device-specific driver 550 may correspond tothe HCA device-specific driver for providing an abstract interface toall of the initialization, configuration and control interfaces of anHCA.

The local system 20 may also communicate with one or more remote systems150, 170 and 30, including I/O units and I/O controllers (and attachedI/O devices) which are directly attached to the switched fabric 100′(i.e., the fabric-attached I/O controllers) using a Virtual Interface(VI) architecture in compliance with the “Virtual Interface (VI)Architecture Specification, Version 1.0,” as set forth by Compaq Corp.,Intel Corp., and Microsoft Corp., on Dec. 16, 1997. NGIO and VIarchitectures support asynchronous data transfers between two memoryregions, typically on different systems over one or more designatedchannels of a data network. Each system using a VI architecture maycontain work queues formed in pairs including a send queue and a receivequeue in which requests, in the form of descriptors, are posted todescribe data movement operation and location of data to be moved forprocessing and/or transportation via a NGIO switched fabric. The VISpecification defines VI mechanisms for low-latency, high-bandwidthmessage-passing between interconnected nodes connected by multiplelogical point-to-point channels. Other architectures such as Infinibandmay also be used to implement the present invention.

In such data networks, NGIO, VI and Infiniband hardware and software maybe used to support asynchronous data transfers between two memoryregions, often on different systems. Each system may serve as a source(initiator) system which initiates a message data transfer (message sendoperation) or a target system of a message passing operation (messagereceive operation). Each system may correspond to a multi-processorsystem including multiple processors each capable of processing an I/Ocompletion on a different shared resource (such as work queues or othermemory elements associated with a given hardware adapter). Examples ofsuch a multi-processor system may include host servers providing avariety of applications or services, and I/O units providing storageoriented and network oriented I/O services.

RDMA operations allow data transfers between memory regions on differentsystems. The RDMA requests are formatted into RDMA elements and invokean appropriate data transfer service. An RDMA element may define thesize of RDMA data buffers and their location in memory. The RDMAelements may also reference multiple local data buffers, but aregenerally limited to a single remote data buffer.

FIG. 6 shows one example embodiment of how data may be transferredbetween a local system 20 and a remote system 30. For ease ofillustration, FIG. 6 does not show the interconnection fabric 100between the local system 20 and the remote system 30. The local system20 may include data buffers 22, 24 and 26 that can be of varying sizesor can be of equal sizes. The data buffers 22, 24 and 26 may be separateor contiguous data buffers. The data buffers 22, 24 and 26 may beprovided within the main memory 206 or they may be provided virtuallyanywhere within the local system 20. An RDMA element 28 may be providedfrom a user and reference local RDMA buffers 23, 25 and 27, which inturn correspond to the data buffers 22, 24 and 26, respectively. TheRDMA element 28 is also associated with the data buffer segment 50provided within the remote system 30. As discussed above, hardware orapplication software may restrict the maximum size of data (hereafteralso referred to as maximum transfer size or capacity) that may besupported for a single data transfer between the local system 20 and theremote system 30. That is, the architecture may only be able to supporta 4 gigabyte data transfer although the amount of data desired to betransferred may exceed 4 gigabytes.

The present invention may coalesce (or merge) small RDMA requests into asingle transfer operation so as to increase the system performance. Thepresent invention may also divide large RDMA requests into multipleoperations to support hardware and software limitations. A combined RDMArequest optimizes system usage by providing a single function call inplace of multiple calls and allows the underlying hardware and softwareto perform the transfer more efficiently. That is, as each call requiresa predetermined amount of set-up time as well as other overhead,excessive/redundant set-up times and overhead can be minimized byminimizing the number of calls made.

More specifically, FIG. 7 shows an exemplary embodiment of how data maybe coalesced or divided in accordance with the present invention. Theoperations may take place within an RDMA manager 60 of the local system20. The RDMA manager 60 may be provided within the host-fabric adaptersoftware stack although the RDMA manager 60 may be provided virtuallyanywhere within the local system 20 provided it may communicate with thebuffer regions. The present invention may be performed by software thatorganizes and appropriately transfers that data between the local system20 and the remote system 30. The software to perform this method may beprovided on any number of tangible storage mediums, including but notlimited to, CD-ROM, diskettes, tapes, etc. The software may also bedownloaded from another computer network, downloaded from the Internetor World Wide Web transferred by a wireless communication network, orany other viable method.

As shown in FIG. 7, an RDMA request 62 is initially provided. The RDMArequest 62 may be formatted into RDMA elements 64, 66 and 68 which inturn correspond to data within the local memory buffers 70, 72 and 74.The local memory buffers 70, 72 and 74 may correspond with the databuffers 22, 24 and 26 shown in FIG. 6. Each RDMA request 62 may beassociated with a REQUEST STATE register or flag that indicates thestate of the current request (i.e, whether it has been fulfilled ornot).

The network architecture may use descriptors to notify the respectivehardware of the data transfer requests. The descriptors specify theremote memory buffer or region 50 (i.e., REMOTE) within the remotesystem 30 to which the data will be transferred. One data segment may beprovided for each local memory buffer (i.e., LOCAL). For the exampleshown in FIG. 7, the RDMA element 64 is associated with the local memorybuffer 70, the RDMA element 66 is associated with the local memorybuffer 72, and the RDMA element 68 is associated with the local memorybuffer 74. The RDMA manager 60 associates each of the local memorybuffers 70, 72 and 74 with a descriptor. Each descriptor in turn isassociated with a different data transfer operation from the localsystem 20 to the remote system 30. In this example, the RDMA manager 60associates the data within the local memory buffer 70 with the firstdescriptor 80 which in turn corresponds with the first transferoperation 84. The example local memory buffer 70 includes less data thanthat which is capable of being transferred to the remote system 30 in asingle transfer operation. In other words, the amount of data is lessthan the maximum transfer capacity (i.e., max RDMA transfer size) of thesystem. The RDMA manager 60 then determines whether any additional databeyond that which in the local memory buffer 70 can be transferred inthe first transfer operation 84. This decision may be based on themaximum transfer capacity of the system and the amount of data alreadyallocated to that data transfer operation. In this example, becauseadditional data can be transferred, the RDMA manager 60 associates aportion 72A of the local memory buffer 72 with the first transferoperation 84. The portion 72A corresponds to an amount of data that whenadded with the amount of data in the local memory buffer 70, does notexceed the maximum transfer capacity of the system. The first descriptor80 is thus associated with data from the local memory buffer 70 and aportion 72A of the local memory buffer 72. As discussed above, theamount of data that may be transferred in the first transfer operation84 depends on the maximum transfer capacity of the architectureincluding the transfer capacity of the local system, the remote systemand the supporting architecture of the network, such as the NGIO, the VIand/or the Infiniband architecture.

Because RDMA operations generally register the descriptors with thehardware, a limited number of descriptors may be available to thesystem. A large RDMA request may require more than one descriptor. Inorder to support RDMA operations that require more descriptors than areavailable, the present invention may maintain the current state of arequest (i.e., REQUEST STATE) in the RDMA manager 60. REQUEST STATE ismaintained to allow the system to divide a single RDMA request intomultiple descriptors and allow continuation of RDMA requests whenadditional resources become available. As discussed above, the presentinvention may format the descriptors to transfer up to the maximum RDMAtransfer size and update the REQUEST STATE to indicate the portion ofdata remaining to be transferred.

The RDMA manager 60 may maintain a pointer at the respective RDMAelements and/or buffer to indicate what data is associated with whichdescriptor. In the above-described example, the first descriptor 80 isassociated with data within the local buffer 70 and the portion 72A ofdata within the local memory buffer 72. In order to determine what data(i.e., which buffer) has been associated with a descriptor and what datahas not been associated with a descriptor, the RDMA manager 60 maintainsa pointer (or counter) CURRENT ELEMENT that references the appropriateRDMA element (and/or buffer) indicating which data has been associatedwith a descriptor and which data has not been associated with adescriptor. A pointer (or counter) OFFSET may identify a location withinthe local memory buffer indicating which data in the buffer has beenassociated with a descriptor and which data has not been associated witha descriptor. Stated differently, the pointer (or counter) CURRENTELEMENT and OFFSET are used to identify data within the local memorybuffers 70, 72 and 74.

In the FIG. 7 example, the pointer OFFSET is used to distinguish theportion 72A associated with the first descriptor 80 and the portion 72Bthat has not yet been associated with a descriptor. After determiningthat the amount of data for the first transfer operation 84 is as muchas the system will allow (or desire), the portion 72B of data within thelocal memory buffer 72 is associated with a second descriptor 82. Thesecond descriptor 82 corresponds to a second transfer operation 86 fromthe local system 20 to the remote system 30. The amount of data withinthe portion 72B is less than the maximum transfer capacity of thesystem. Therefore, the RDMA manager 60 determines whether any more datacan be associated with the second transfer operation 86. Again, this maybe accomplished by referencing the maximum transfer capacity of thesystem. In this example, all the data within the local memory buffer 74can also be transferred in the second transfer operation 86 because thecombination of data within the portion 72B and the data within the localmemory buffer 74 is less than the maximum transfer capacity of thesystem. Accordingly, the portion 72B and the buffer 74 are associatedwith the second descriptor 82 and the second transfer operation.

The present invention may associate data within the local buffers witheach of the descriptors 80 and 82 prior to any transfer operationoccurring. In this situation, the RDMA manager 60 may obtain a totaltransfer count indicating a number of transfers that will be necessaryto satisfy the RDMA request 62. The RDMA data manager 60 may alsodetermine the first descriptor 80, perform a first data transferoperation 84, then determine a second descriptor 82 and subsequentlyperform a second data transfer operation 86. In other words, the presentinvention may determine the second descriptor 82 (or any subsequentdescriptors) prior to, during, or subsequent to the first data transferoperation 84.

If multiple RDMA elements reference different local memory buffers butare destined for contiguous, remote memory regions, the data transfersmay be combined (i.e. coalesce) into as few transfers as possible.However, if a coalesced or single request is larger than the maximumRDMA transfer size or requires more data segments than those allocatedfor a descriptor, then the request may be divided among multipledescriptors. This allows optimal usage of hardware and descriptorresources.

The above-described example embodiments have been described with respectto transferring data to a remote memory. The present invention is alsoapplicable to transferring data from a remote system 30 and into a localsystem 20. In this type of operation, the descriptors 80 and 82 may beformatted such that they provide the data from the remote memory 50 intothe local memories 70, 72 or 74. Using the example embodiment of FIG. 7,the RDMA manager 60 desires to fill the local buffers 70, 72 and 74 withdata from the remote memory 50. The RDMA manager 60 formats thedescriptor 80 such that the local memory 70 and portion 72A of localmemory 72 are associated with the first transfer operation 84 from theremote memory 50 to the local system 20. This is accomplished by knowingthe maximum transfer capacity of the system and by knowing the capacityof each of the local memories 70, 72 and 74. The RDMA manager 60likewise associates portion 72B of local memory 72 and local memory 74with the second descriptor 82 and the second transfer operation 86because this amount of data does not exceed the maximum transfercapacity of the system. In similar manner as described above, the RDMAmanager 60 may utilize the descriptors 80 and 82 by using one pointerCURRENT ELEMENT and OFFSET to properly reference the local memorybuffers that the data should be placed in after being transferred fromthe remote memory buffer 50.

FIG. 8 shows another example embodiment of how data within local memorybuffers 200 and 202 may be divided and coalesced into two descriptors 80and 82 for two independent data transfer operations. Local memorybuffers 200 and 202 may be similar to the local memory buffers 70, 72and 74 of FIG. 7. In this example, the amount of data within localmemory buffer 200 exceeds the maximum data transfer capacity of thesystem so the RDMA manager 60 divides the data into a first portion 200Aand a second portion 200B. The first portion 200A is associated with thefirst descriptor 80 and the first transfer operation 84. The firstportion 200A does not exceed the maximum transfer capacity of thesystem. The second portion 200B is associated with the second descriptor82 and the second transfer operation 86. In this example, the datawithin the local memory buffer 202 may be coalesced (or combined) withthe portion 200B provided the total amount of data is less than themaximum transfer size. Hence, the data within local memory buffer 202 isassociated with descriptor 82 and the second transfer operation 86.

FIG. 9 shows a brief overview of an example algorithm used by thepresent invention. This algorithm may be performed by the software ofthe networked system and is merely illustrative of the presentinvention. That is, other algorithms of performing the invention arealso within the scope of the present invention.

FIG. 9 shows that an initial request for RDMA services is made (block300). This request may be either a read or write operation (block 302).The request is initialized in block 304. Then, in block 306, the chainof RDMA elements is determined. The system then performs the coalescingand dividing algorithm as discussed above (block 308). Blocks 304, 306and 308 are repeatedly performed to count the total transfer count orthe number of transfers needed to complete the RDMA request. Eachtransfer operation may require one descriptor as discussed above. Theformation of RDMA elements 64, 66 and 68 may be achieved in block 306.Using the algorithm, the respective local memory buffers 70, 72 and 74may be coalesced and/or divided in block 308 so as to determine thetotal transfer count and to determine the descriptors. This may involvecounting the number of completed transfers as well as using the pointersCURRENT ELEMENT and OFFSET as discussed above.

In block 310, the first descriptor 80 and the second descriptor 82 maybe posted. The formatted descriptors may then be placed in the workqueue and the RDMA data transfer may be performed in block 312. Afterthe data transfer for the first descriptor 80, the system may update thenumber of completed transfers in block 314 and check whether the RDMArequest is complete in block 318. If additional data transfers areneeded (i.e., the system has not performed the total transfer count)then block 310 may post the next descriptor, such as the seconddescriptor 82, and perform the appropriate RDMA data transfer in block312. Blocks 314, 316 and 318 are likewise performed again. Uponcompletion of this loop, a signal may be output in block 320 to signalthat all of the data of the initial request has been transferred.

The following algorithm may be used to format a single descriptor andincludes the coalescing and dividing operation. This algorithmcorresponds to the coalesce/divide elements routine 308 shown in FIG. 9and its relation with block 306. This algorithm is merely illustrativeand is not meant to be limiting of the invention. That is, otheralgorithms are also within the scope of this invention.

Format Descriptor ( Request, Descriptor ) {  Initialize the descriptor Current Transfer Size = 0  Current Element = Request State CurrentElement  // Walk the chain of RDMA elements in an attempt to coalescethe elements.  for each data segment allocated in the Descriptor  {   //Determine how much data still needs to be transferred for the   //current RDMA element.   Element Size Remaining = Current Element ByteCount − Request State Offset   // Determine how much can be transferredby the current descriptor.   Transfer Size Remaining = Max RDMA Transfer− Current Transfer Size   // Set the starting address for this datasegment of the transfer.   Descriptor Data Segment Local Address =   Current Element Local Address + Request State Offset   // See if weneed to divide the RDMA element into multiple requests.   if ( ElementSize Remaining > Transfer Size Remaining )   {    // We will needanother descriptor to complete this RDMA element.    // Place what wecan in this descriptor.    Descriptor Data Segment Length = TransferSize Remaining    // Update the state of the request and return.   Request State Offset = Request State Offset + Transfer Size Remaining   exit for loop   }   // Add the rest of the RDMA element to thedescriptor.   Descriptor Data Segment Length = Element Size Remaining  Current Transfer Size = Current Transfer Size + Element Size Remaining  if there is not another RDMA element   {    // Update the RequestState to signal that we are done. The completion    // routine willcheck this to see if more descriptors need to be posted.    RequestState Current Element = NULL    exit for loop   }   Update the RequestState to reference the next element   Request State Offset = 0   // Seeif we can coalesce the next RDMA element.   if we cannot coalesce thenext element   {    exit for loop   }  }  Complete formatting thedescriptor }

The following algorithm provides one example of how to coalesce/dividethe data within the local memory buffers into different data transferoperations. This algorithm is merely illustrative and is not meant to belimiting of the invention. That is, other algorithms are also within thescope of this invention.

boolean FormatRdmaRequest( ) {  // Walk the buffers in an attempt tochain together smaller requests.  while we still have buffers and thetotal transfer length does not exceed the maximum     supported  {  Determine how much data still needs to be transferred for the  current buffer and how much can be transferred by the current  request.   Add the buffer to the descriptor.   // See if we need tosplit the RDMA object into multiple requests.   if the current buffer istoo large to transfer in its entirety   {    Place what we can in thisdescriptor.    Update the size of the request.    Save the offsets.   exit the while loop   }   Add the rest of the buffer to thedescriptor.   Get a reference to the next buffer in the chain.   Resetthe current offset since we are starting a new buffer.  }  Set totalsize of transfer and save the state of the transfer  Return TRUE if thetransfer is complete }

The present invention may provide unique advantages based on its RDMAcoalescing and dividing algorithm. The present invention may abstractthe details of NGIO, VI and Infiniband descriptors away from developersto thereby decrease the application and driver development time. Thepresent invention may also optimize the usage of hardware by coalescingmultiple RDMA requests into a single data transfer. The presentinvention may further provide a mechanism to support RDMA transferslarger than those supported by the underlying hardware implementations.The present invention may provide an algorithm to coalesce and divideRDMA requests over NGIO, VI and Infiniband architectures. The presentinvention may also maintain and check the state of RDMA requests,allowing for better system resource allocation and usage by permittingdescriptor reuse.

Several example embodiments of the present invention are specificallyillustrated and/or described herein. However, it will be appreciatedthat modifications and variations of the present invention are coveredby the above teaching and are within the preview of the appended claimswithout departing from the spirit and intended scope of the invention.For example, while the present invention has been described withreference to NGIO, VI and Infiniband architecture, the various aspectsof the present invention are applicable to other types of networks thatinclude data transfers between a local computer system and a remotecomputer system.

1. A method comprising: if an amount of data located in one or morelocal memory buffers in a local computer system associated with one ormore remote direct memory access (RDMA) requests does not exceed amaximum transfer size for a single memory to memory transfer operationover a data network to one or more remote memory buffers in a remotecomputer system: associating the data located in the one or more localmemory buffers with a first transfer operation over the data network tothe one or more remote memory buffers in the remote computer system; ifan amount of data associated with the first transfer operation has notreached the maximum transfer size: associating at least a portion ofdata, otherwise to be associated with one or more subsequent transferoperations associated with the one or more RDMA requests, with the firsttransfer operation instead of with the one or more subsequent transferoperations, the portion of the data being located in one or more othermemory buffers in the local computer system, the one or more subsequenttransfer operations to be over the data network to the remote memorybuffer in the remote computer system, the one or more local memorybuffers being separated from the one or more other memory buffers; andif the one or more local memory buffers comprises a plurality ofdifferent local memory buffers and the one or more remote memory bufferscomprises contiguous remote memory buffers: minimizing a number oftransfer operations employed to transfer the data that is located in theplurality of different local memory buffers so as to conform to aminimum number required to transfer the amount of data located in theplurality of different local memory buffers to the contiguous remotememory buffers.
 2. The method of claim 1, wherein: if the amount of datalocated in the one or more local memory buffers exceeds the maximumtransfer size: associating part of the data located in the one or morelocal memory buffers with the first transfer operation; and associatingone or more other portions of the data located in the one or more localmemory buffers with the one or more subsequent transfer operations. 3.The method of claim 1, further comprising: maintaining pointers to RDMAelements to indicate associations of descriptors with data to betransferred via the transfer operations and other data that has not yetbeen associated with any descriptor.
 4. The method of claim 3, wherein:the transfer operations comprise RDMA operations that compriseregistration of the descriptors with hardware of the local computersystem; and the method comprises maintaining a current RDMA requeststate so as to permit division of a RDMA request into multipledescriptors.
 5. The method of claim 4, wherein: the RDMA elements definesizes and locations of the local and remote memory buffers.
 6. Themethod of claim 5, wherein: the RDMA request is formatted into the RDMAelements.
 7. An apparatus comprising: an RDMA (remote direct memoryaccess) manager operable to perform operations comprising: if an amountof data located in one or more local memory buffers in a local computersystem associated with one or more remote direct memory access (RDMA)requests does not exceed a maximum transfer size for a single memory tomemory transfer operation over a data network to one or more remotememory buffers in a remote computer system: associating the data locatedin the one or more local memory buffers with a first transfer operationover the data network to the one or more remote memory buffers in theremote computer system; if an amount of data associated with the firsttransfer operation has not reached the maximum transfer size:associating at least a portion of data, otherwise to be associated withone or more subsequent transfer operations associated with the one ormore RDMA requests, with the first transfer operation instead of withthe one or more subsequent transfer operations, the portion of the databeing located in one or more other memory buffers in the local computersystem, the one or more subsequent transfer operations to be over thedata network to the remote memory buffer in the remote computer system,the one or more local memory buffers being separated from the one ormore other memory buffers; and if the one or more local memory bufferscomprises a plurality of different local memory buffers and the one ormore remote memory buffers comprises contiguous remote memory buffers:minimizing a number of transfer operations employed to transfer the datathat is located in the plurality of different local memory buffers so asto conform to a minimum number required to transfer the amount of datalocated in the plurality of different local memory buffers to thecontiguous remote memory buffers.
 8. The apparatus of claim 7, wherein:if the amount of data located in the one or more local memory buffersexceeds the maximum transfer size, the RDMA manager is operable to:associate part of the data located in the one or more local memorybuffers with the first transfer operation; and associate one or moreother portions of the data located in the one or more local memorybuffers with the one or more subsequent transfer operations.
 9. Theapparatus of claim 7, wherein the RDMA manager also is operable to:maintain pointers to RDMA elements to indicate associations ofdescriptors with data to be transferred via the transfer operations andother data that has not yet been associated with any descriptor.
 10. Theapparatus of claim 9, wherein: the transfer operations comprise RDMAoperations that comprise registration of the descriptors with hardwareof the local computer system; and the RDMA manager also is operable tomaintain a current RDMA request state so as to permit division of a RDMArequest into multiple descriptors.
 11. The apparatus of claim 10,wherein: the RDMA elements define sizes and locations of the local andremote memory buffers.
 12. The apparatus of claim 10, wherein: the RDMArequest is formatted into the RDMA elements.
 13. Computer-readablestorage device including program instructions to be executed by acomputer processor for performing a set of operations comprising: if anamount of data located in one or more local memory buffers in a localcomputer system associated with one or more remote direct memory access(RDMA) requests does not exceed a maximum transfer size for a singlememory to memory transfer operation over a data network to one or moreremote memory buffers in a remote computer system: associating the datalocated in the one or more local memory buffers with a first transferoperation over the data network to the one or more remote memory buffersin the remote computer system; if an amount of data associated with thefirst transfer operation has not reached the maximum transfer size:associating at least a portion of data, otherwise to be associated withone or more subsequent transfer operations associated with the one ormore RDMA requests, with the first transfer operation instead of withthe one or more subsequent transfer operations, the portion of the databeing located in one or more other memory buffers in the local computersystem, the one or more subsequent transfer operations to be over thedata network to the remote memory buffer in the remote computer system,the one or more local memory buffers being separated from the one ormore other memory buffers; and if the one or more local memory bufferscomprises a plurality of different local memory buffers and the one ormore remote memory buffers comprises contiguous remote memory buffers:minimizing a number of transfer operations employed to transfer the datathat is located in the plurality of different local memory buffers so asto conform to a minimum number required to transfer the amount of datalocated in the plurality of different local memory buffers to thecontiguous remote memory buffers.
 14. The device of claim 13, whereinthe set of operations further comprises: if the amount of data locatedin the one or more local memory buffers exceeds the maximum transfersize: associating part of the data located in the one or more localmemory buffers with the first transfer operation; and associating one ormore other portions of the data located in the one or more local memorybuffers with the one or more subsequent transfer operations.
 15. Thedevice of claim 13, wherein the set of operations further comprises:maintaining pointers to RDMA elements to indicate associations ofdescriptors with data to be transferred via the transfer operations andother data that has not yet been associated with any descriptor.
 16. Thedevice of claim 15, wherein: the transfer operations comprise RDMAoperations that comprise registration of the descriptors with hardwareof the local computer system; and the set of operations furthercomprises maintaining a current RDMA request state so as to permitdivision of a RDMA request into multiple descriptors.
 17. The device ofclaim 16, wherein: the RDMA elements define sizes and locations of thelocal and remote memory buffers.
 18. The device of claim 16, wherein:the RDMA request is formatted into the RDMA elements.